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ABSTRACT 


Recently  a  new  statistical  methodology^  developed  over  the  last  3 
decades,  has  become  available  to  practitioners.  This^jpethodology  is 


,  * 

called  "ranking  and  selection*theory .  In  this  article  we  review  proce- 

\  :  S 

dures  for  completely  ranking  a  set  of  populations  (from  "best”,  "second 
best",  etc.,  down  to  "worst”),  givei,new  tables  needed  to  implement  these 
orocedures,  and  consider:  several  practical  examples  using  real  data. 
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VW^ECT£o 

1.  INTRODUCTION 

Since  1954  (Bechhofer  (1954))  a  new  statistical  methodology  has 
been  developing.  This  methodology  is  called  "ranking  and  selection" 
theory,  and  has  recently  become  accessible  to  practitioners  (Gibbons, 
Qlkin,  and  Sobel  (1977),  Dudewicz  (1980)),  In  this  section  we  review  the 
problem  of  completely  ranking  a  set  of  populations  (from  "best",  "second 
best",  etc.,  down  to  "worst").  New  tables  needed  to  implement  these 
procedures  are  given  in  Section  5,  with  a  discussion  of  their  construction 
in  Section  3.  Practical  examples  using  real  data  are  analyzed  in  Section 
4.  These  examples  should  aid  researchers  in  many  fields  in  proper  use 
of  this  new  methodology. 
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2.  PROCEDURE 


12  FORMULATION 

Let  ttj  ,tt2>  •  •  •  .rr^  represent  k(  >  2)  independent  sources  of  random 

variables.  Assume  observations  from  t.  are  normallv  distributed 

i 

2  .22 
N(y^,a^)  ( 1  <  i  <  k)  with  means  p  ,...,p  and  variances  all 

unknown.  Assume  {Indifference  Zone  Formulation)  that  the  goal  is  to 

completely  rank  the  populations  from  that  with  the  largest  mean  U^-j 

down  through  that  with  the  smallest  mean  (where 

denote  the  ordered  means)  in  such  a  way  that  the  probability  of  correct 
ranking  P(CR)  satisfies 

P(CR)  >  P"  whenever  -  u j- j _ j  -j  ^  6y  (j«2,...,k)  (1) 

where  P”  and  6 ^ *  -  •  * » <5^  O/lt!  <  P“  <  1 ; 0  <  are  specified  in 

advance  by  the  experimenter.  It  has  been  shown  by  Dudewicz  and  van  der 
Meulen  (1980)  that  this  is  guaranteed  by  the  following  procedure 

VCR-IZ)* 


PROCEDURE  (T„„(CR-IZ)  .  Take  an  initial  sample  X. X.  of  size 
-  DD  r  1 1  10q 

n^(  >  2)  from  and  define 


Xi(nQ) 


V”o 


«ij  -  W>  ,(»o  -  ')• 


where  5::  >  0  is  arbitrary  and  h  >  0  is  the  unique  solution  of  the 
equation 
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P"  -  P[Y  j  < Y ^  +  5"h/6::.,Y2<Y3  ♦  5”h/5:: . Y,^  <  \  +  5"h/S=-] 


(4) 


where  Yj,...,Y^  are  ’ndependent  Student's  -  t  random  variables  each 
with  ng  -  1  degrees  of  freedom.  (Notice  that 


P[Yj<Y7  +  5"h/6",Y2<  Y3  +  5”h/6",Y3<  Y4  +  S«h/ . . . , Yfc_ ,<  Yfe  +  6gh/6«] 


jm  ^4+64h/6"  jy3+63h/s:: 


-QO  —  CO 


F  (y2  +  «gh/6*e)f  (y2)f  (y3)  fn„(yk-l)fn„(yk)dy2dy3  *•*  dyk-ldyk  (5) 


0  *•  0 


where  F  (•)  and  f  (•)  are  respectively  the  distribution  and  density 
n0  n0 

function  of  a  Student's  -  t  random  variable  with  nQ-l  degrees  of  freedom.) 

In  (3),  [y]  denotes  the  smallest  integer  >  y.  Take  iu  -  n^  additional 

observations  X.  X.  from  ir.  and  define 

i,n_+l  in.  i 

0  l 


n. 

l 


X.  =>  ]  a..X.  . 

l  if,  ij  ij 


j  =  I 


(6) 


(i  ■  1 , . . . ,k) .  (Here  the  ' s,  j  =  1 , . .  .  ,n^,  1  <  i  <  k,  are  chosen  so 


that 


n. 

l 


Y  a.  .  »  1 ,  a.  . 

it,  U  il 


j-1 


ni 

a.  ,  and  s?  J  a? .  ■  (3::/h)^. 
in0  i  j-l  1J 


(7) 


To  be  specific,  take  the  positive  radical  solution  from  a.  .  ■  ... 

1,n0 

»  a.  .)  Finally,  assert  (for  i  =  l,...,k)  that  the  population  which 
in. 

l  ^  cr  -  - 

yielded  Xr.,  has  mean  ur. ,  (where  Xri,<  ...  <  Xrt  ,  denote  X.j.-.jX.  in 
Li]  UJ  LH  LKJ  1  * 

numerical  order) . 


If  one's  indifference  zones  are  fully  symmetric  in  that 

<5v  »  ...  =  6"  *  6"  (say),  then  the  h  »  h  ,  (P:i)  to  be  used  is  the  unique 
z  k.  nQ » k 

solution  of 

P"  -  P(Yj<Y2  +  h,Y2<Y3  +  h,Y3<Y4  +  h . \_!<Yk  +  (8) 

where  Y.,...,Y^  are  independent  Student's  -  t  random  variables  each  with 
ng  -  1  degrees  of  freedom.  In  what  follows,  we  suppose  (T^(CR-IZ)  is 
always  used  with  5^  »  ...  •  <S"  *  5";  the  corresponding  h,  which  is  denoted 
by 


h  ‘  h„0,k(p!:’62/6“ . Sk/6i:> 


where  6v/<5"  »  1  (i  «  2,...,k),is  tabulated  in  Section  5. 


PPF  FORMULATION 

In  the  setting  considered  above,  another  requirement  is  often  desired 
by  experimenters  :  that  they  be  100P::  %  sure  the  ranking  they  ultimately 
state  is  correct  up  to  interchanges  of  populations  whose  true  means  differ 
by  6::  or  less.  Thus,  the  Pvefarred  Population  Formulation  (PPF)  of  the 
complete  ranking  problem  has, as  its  goal , statement  of  a  complete  ranking 
of  the  populations  in  such  a  way  that 

P(CR)  >  P=:  (10) 

for  all  possible  parameter  configurations,  where  P”  (1/k!  <  P::  <  1)  is 
specified  in  advance  by  the  experimenter  and  event  "CR"  is  considered 
to  occur  if  the  order  specified  is  correct  or  can  be  made  correct  by  one 
or  more  interchanges  of  assertions  involving  populations  whose  true  means 
differ  by  at  most  6:;  (5::  >  0)  . 
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Now  it  is  shown  in  Dudevicz  and  van  der  Meulen  (1980)  that  procedure 
lTpQ(CR  -  IZ)  with  <5"  ■  ...  =  6"  -  6::  given  above  satisfies  P(CR)  >  P:: 
with  this  new  definition  of  event  CR.  Thus  the  following  procedure 
achieves  the  goal. 

PROCEDURE  (CR  -  PPF) .  Choose  5“  *  ...  =  S"  =*  6"  and  proceed  according 

to  procedure  (J'^(CR-IZ). 

3.  CONSTRUCTION  OF  TABLES  FOR  ^(CR-IZ)  AND  (T^CCR-PPF) 

In  order  to  tabulate  h  which  solves  equation  (8),  for  each  n^ 
(10,15,20,25,30)  of  interest  we  called  a  Monte  Carlo  evaluation  routine 
SIMUL,  given  below  in  Figure  1 ,  in  a  pinch  process  attempting  to  converge 
on  the  root. 

For  computational  efficiency,  subroutine  SIMUL  does  evaluation  of  the 
P(CR)  simultaneously  for  each  of  k  ■  2, 3,..., 25  and  also  simultaneously 
at  144  h  values,  with  each  h  value  chosen  so  as  to  be  closest  to  the  root 
of  the  corresponding  (8)  when  P"  =  . 75 , . 80 , . 85 , . 90, .95 , . 975  and  k.  * 

2, 3,..., 25.  SIMUL  looks  at  10,000  samples  of  k  Student's  -  t  random 
variables  Yj,...,Y^  with  n^  -  1  degrees  of  freedom.  These  are  constructed 
from  kn^  independent  standard  normal  random  variables  generated  using 
the  Box-Muller  transformation  to  normality  on  pseudo-random  numbers  from 
generator  UNI  with  seeds  IX  =  524,287  and  JX  *  654,345,465;  this  generator, 
which  has  a  period  of  2^  _  2^  ^  7  x  10^,  has  been  found  to  have  good 
properties  in  extensive  testing  by  Dudewicz  and  Ralley  (1981).  Then  SIMUL 
reports  the  proportion  of  the  10,000  samples  where  a  completely  correct 
ranking  was  achieved,  in  the  sense  that 
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Y  <Y_  +  h  <  Y ,  +  2h<  ...  <  Y  ,  +  (k-2)h  <  Y,  +  (k-l)h.  (11) 

123  k- 1  k 

These  computer  runs  were  preceded  by  a  Monte  Carlo  tabulation  of  P(CR) 
as  a  function  of  h  =  0.0  (0.1)  5.1,  from  which  intervals  in  which  the 
true  root  of  (8)  lies  were  determined  for  use  in  the  later  pinch  process 
just  described,  and  were  followed  by  supplemental  runs  where  needed 
i.for  additional  accuracy,  or  where  the  preliminary  interval  used  did  not 
in  fact  contain  the  true  root) .  These  runs  used  random  number  generator 
RANDOM  with  seed  INT  -  524,287.  This  generator  was  recommended  by  Dudewicz 
(1976)  after  preliminary  investigations  for  goodness  and  speed.  The  re¬ 
sults  of  the  pinch  process  are  given  in  the  tables  of  Section  5,  where 
for  each  value  of  k,  a  separate  table  is  given  containing  as  entries  the 
h  values  corresponding  to  a  fixed  choice  of  P::  and  n^.  E.g.  for  k  “  10, 

P"  *  0.975,  and  »  20,  ona  finds  h  =  4.29. 

The  tables  also  include  values  of  h  for  the  case  n^  =*  =®  (used  when 
one  knows  the  variances  o::  the  populations) ,  calculated  by  a  Monte  Carlo 
simulation  procedure  similar  to  the  one  described  above  but  with  use  of 
standard  normal  random  variables  instead  of  Student's  -  t  random  variables. 
In  the  latter  case  the  pseudo-random  numbers  were  generated  usiijg  genera¬ 
tor  RANDOM.  These  values  also  facilitate  interpolation  for  >  30. 

The  tables  also  list  values  of  h  for  the  cases  m  2(1)9.  As  values 
of  >  10  are  recommended  for  use  in  usual  practice,  fewer  decimals  are 
reported  in  the  tables  for  n^  <  10.  Fewer  decimals  were  also  reported  for 
P"  "  .99,  but  there  because  (see  the  accuracy  analysis  below)  further 
decimals  would  be  inaccurate.  These  table  parts  were  obtained  from  the 
Monte  Carlo  tabulation  of  P(CR)  as  a  function  of  h  -  0.0(0. 1)5.1  (supplemen¬ 
ted  by  additional  evaluations  for  cases  where  h  >  5.1). 
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SUBROUTINE  SIMUUHM, PCR ,NO) 

DIMENSION  Y(25) .11(750) ,X(7S0) , NCR (24, 6) ,PCR(24,6) ,HM(24,6) 

IX=524287 

JX=654345465 

CALL  RSTART(iX.JX) 

DO  2  Ll=l ,24 
DO  2  L2=l , 6 

2  NCR(L1 , L2 ) =0 
N=0 

Nl=((NO*25+l)/2)*2 

3  DO  50  Ml=l ,N 1 

50  U(!il)=UNI(0) 

N2=(NO*25)/2 
DO  51  M2=1,N2 
M2A=2*M2-1 
M2B=M2A+1 

X(M2A)=SQRT( -2 .*ALOG(U(M2A) ) )*COS (2 .*3 ■ 141593~U(M2B) ) 

51  X (M2B )  =SQR'f  ( -  2 .  *ALOG  (U (M2A ) ) )*S IN ( 2 . *3 . 1 4 1 59  3*U (M2B  J ) 

IF  (N2.LT. UNO*25.)/2.))  X(Nl-l)=SQRT(-2 .*AL0G(U(N1-I ) ) )* 
+COS(2.*3. 141593*U(N1 ) ) 

N3-NO- 1 

DO  52  -13=1,25 

SUM=0 

DO  53  J4=1,N3 

53  SUM=SUh+X( (J3-l)*NO+l+J4)**2 

52  Y(J3)=X(U3-l)*NO+l)/SQRT(SUM/N3) 

DO  5  11=1,24 

DO  5  12=1,6 
H=HM(I1 ,12) 

K=1 

6  IF  ((Y(K)+(K-1. )*H).GE. (Y(K+1 )+K*H) )  GOTO  5 
K=K+1 

IF  (K.LE.I1)  GOTO  6 
NCR( I 1 , I2)=NCR( I 1 , I2)+l 
5  CONTINUE 
N=N+1 

IF  (N.LT. 10000)  GOTO  3 
DO  7  L3=l , 24 
DO  7  L4=l , 6 

7  PCR(L3 ,L4 )=NCR(L2 , L4)/ 10000 . 

RETURN 

END 


Figure  !~.  Subroutine  SIMUL 


"  Thanks  are  due  to  Mr.  P.  Darius  for  preparin'’  the  printout  used  in 


Figure  1 . 


ACCURACY  ANALYSIS 


A  number  of  checks  of  accuracy  of  Che  computations  were  carried  out, 
and  will  now  be  described  briefly.  A  P(CR)  computation  using  the  proce¬ 
dure  described  above  is  accurate  (with  95  %  probability)  to  within 
+  2  / P" ( 1 -P") / 1 OVOOO ,  tabulated  in  Table  1. 

Table  1. 


p:: 

2  kP::(l-P::)/'10,000 

.50 

.010 

.  75 

.008 

.80 

.008 

.85 

.007 

.90 

i 

.006 

.95 

.004 

.975 

.003 

.99 

.002 

Now,  as  a  first  check,  when  h  »  0.0  we  know  theoretically  (from  (8)) 
that  P(CR)  •  1/k!,  which  was  (within  the  Table  1  accuracies  of  the  Monte 
Carlo)  confirmed  in  our  tabulation  of  P(CR)  as  a  function  of  h  *  0.0(0. 1) 
5.1.(E.g.,  when  k-2  and  ng»25  we  found  P(CR)  *  .4967,  which  is  within 
.01  of  the  true  value  ,5000.)  As  a  second  check,  when  k“2  the  problem 
of  a  complete  ranking  is  equivalent  to  the  problem  of  selecting  the  best 
(since  then  the  population  not  selected  must  be  inferior  if  the  selection 
of  the  best  has  been  correctly  made).  For  this  latter  problem,  Dudewicz, 
Ramberg,  and  Chen  (1975)  have  tabulated  h  in  their  Table  4.  Comparison 
shows  our  method  leads  to  values  correct  to  +  2  units  in  the  last  place 
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shown  (84  entries),  _+  3  units  (4  entries),  +  4  units  (1  entry),  +_  5  units 
(1  entry).  In  addition  1  entry  matched  exactly. 

Also  for  k=2,  Bechhofer  (1954)  has  tabulated  h  in  his  table  I  for 
the  case  Oq  «  «.  Again  comparison  shows  our  method  leads  to  values  correct 
to  +  1  unit  in  the  last  place  shown  (2  entries) ,  +  2  units  (2  entries) , 

+  3  units  (1  entry) . (Table  I  of  Bechhofer  (1954)  does  not  have  entries 
for  P»  =  .85,  .975.) 

Finally,  our  tables  should  be  monotone  increasing  in  k  and  P", 
and  monotone  decreasing  in  n^.  This  is  fulfilled  except  for  the  monctoni- 
city  in  n^  where  in  one  case  h,Q  ^(P")  <  h^^  ^(P::)  and  in  5  cases 
h_e  .  (P")  <  hin  (P::) .  This  however  can  be  explained  from  the  accuracy 
analysis  and  the  flatness  of  the  curve  h(n^)  near  larger  values  of  n^. 

To  illustrate  the  accuracy  of  the  tables  as  presented,  let  us  consider 
the  entry  h  »  2.06  when  k  *■  4,  nQ  »  15 ,  P::  *  .  75 .  Hern  the  pinch  process 
yielded  the  results  : 


Table  2. 


h 

P(CR)  Estimated 

2.05000 

.7473 

2.05625 

.7485 

2.06250 

.7518 

2.07500 

.7551 

2.  10000 

.7610 

(though  not  in  this  order).  As  the  estimated  P(CR)  is  good  (probability 


95  X )  to  within  +  .008  when  P::  «  .75,  we  have  established  that  the  true 
value  h  is  less  than  2.10...  but  cannot  distinguish  2.06  from  2,07,  etc. 
(see  the  graph  in  Figure  2  below).  Thus  (on  the  upper  side)  2.06 
is  within  4  units  of  the  true  root  h  (which  cannot  exceed  or  even  equal 
2.10).  If  the  estimated  P(CS)  at  the  h  value  2.07  had  been  greater  chan 
.758,  we  would  then  have  attempted  to  distinguish  another  significant 
digit.  This  analysis  was  carried  through  for  the  k  «*  10  and  k  -  25 
tables,  as  well  as  the  k  «  2  table  (where  -  see  above  -  independent 
confirmation  is  available  from  Dudewicz,  Ramberg,  and  Chen  (1975)).  It 
indicated  an  absolute  error  in  tabled  values  of  usually  <  5  or  6  units 
in  the  last  place  reported,  when  >  3.  (At  n^  ■  2  the  entries  can 
possibly  be  subject  to  much  greater  errors,  but  this  is  not  of  substan¬ 
tial  interest  as  nQ  >  10  is  the  usual  case  for  practical  use.)  As  our 
results  for  the  case  k  =  2  were  in  fact  much  better  than  these  rough 
bounds  would  suggest,  we  believe  these  tables  are  of  such  accuracy  as  can 
be  safely  used  in  practice,  and  do  not  expect  inaccuracy  to  exceed  5 
units  in  the  last  place  reported. 
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Figure  2* 


Illustration  of  Accuracy 


Analysis 


13  - 


4.  EXAMPLES 

Example  1.  Shuttle-Run  Timings.  Daca  on  times  in  the  so-called  shuttle- 
run  (480  meters)  of  62  girls  aged  (by  rounded  off  Larson  age",  as  given 
by  Larson  (1974))  7,  8,  and  9  are  given  below.  (Thanks  for  this  dataset 
are  due  to  Mrs.  M.  Vuylsteke-Wauters ,  co-promotor  at  K.U.  Leuven  of 
E.  Anthonissen  (1981).)  Here  k  =  3  populations  are  present  with  respecti¬ 
vely  27,  17  and  18  observations.  Let  us  choose  n^  »  10  observations  in 
our  first  stage.  If  we  desire  to  have  probability  P::  *  .95  that  our 
ultimate  ranking  is  PPF-correct,  we  will  (from  our  tables)  need  to  use 
h  =  3.19. 

From  the  data  we  find  Xj(10)  »  316.80,  X^OO)  *  280.40,  X^(10)  - 
275.40,  Sj  ■  28.69,  S2  “  23.00,  s^  “  22.35.  If  we  wish  to  be  95  % 

(100  x  P”%)  sure  our  ultimate  ranking  is  correct  up  to  interchanges 
involving  populations  whose  true  means  differ  by  at  most  S::(6::>  0),  we 
will  need  total  sample  sizes  of 


”  Larson  age  is  defined  as  : 

date  of  examination  (decimals)  -  date  of  birth  (decimals) . 

(Three  decimals  are  reported,  being  calculated  from 

(number  of  the  day  in  the  year)  -  1 
365 

For  example,  date  of  examination  :  11th  January  1981  •  81.027 
date  of  birth  :  20th  July  1973  «  73.548 

age  at  time  of  examination  -  7.479  years 

Here  "age  7"  means  a  Larson  age  between  6.95  and  7.95,  l.e.  an  age 
between  6  years  11  2/5  months  and  7  years  11  2/5  months. 


n,  » 
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(3 . 19) 2 (28 . 69) 2 

(3. 19)2(23.00) 2 

(3. 19) 2 (22 . 35) 2 

( 5”) 2 

,  n2  ’ 

( <S:!) 2 

i  n  ^  ® 

(6': ) 2 

(12) 


from  the  three  respective  populations.  Since  we  are  constrained  to 
n^  <  27,  <  17,  n^  <  18,  we  choose  6::  as  small  as  possible  (the 

strongest  resulting  guarantee)  without  violating  these  constraints. 

Thus,  from  (3)  we  find  =  17.80  seconds,  which  yields  nj  «  27,  n^  “  17, 
n,  -  17. 


Solving  for  the  a..'s  in  display  (6)  we  find  for  population  1 


(7  year  olds) , 


taking  a]  E  aljt  -  a]>2  ...  Sj^  and  bj  _  a1>n  ...  -  a1>27, 

that  a |  and  bj  solve  the  system 


C  28 . 69 ) 2  ( 10  a  “I  +  17  bj)  -  ( 1 7 .80) 2/ (3- 19) 2 
10  Bj  +  ■  17  bj  =*  1 . 


03) 


Taking  the  positive  radical  solution  yields  Sj  ■  .0230  and  bj  -  .0412. 

Hence 

Xj  -  (.0230)(3168)  +  ( .0412) (260  +  311  +  ...  +  311)  =  292.97.  (14) 


Similarly  for  population  2  (8  year  olds)  a2  -  a2  j  =  • • •  “  a2  io  *  *®576, 
b2  =  ^2  j j  =  ‘ • *  =  a2  i 7  *  -0605,  X2  *  282.29.  Finally,  for  population 
3  (9  year  olds)  a3  e  a3>,-  ...  -  a3J0  -  .0468,  b3  e  b3>1,  -  ...  -  b^j, 
.0760,  and  X3  -  269.49. 

We  conclude  that  (in  increasing  order  of  speed)  the  groups  are  : 


Fastest  =  9  year  olds 
2n<*  Fastest  **  8  year  olds 
Slowest  -  7  year  olds 
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with  95  Z  confidence,  subject  to  reversals  on  groups  with  true  means 
closer  than  S::  »  17.80. 

Example  2.  Agricultural  Treatments  and  Fertilizers.  Each  year  agricul¬ 
tural  experiments  are  run  at  Heverlee  and  other  locations  (stations)  in 
Belgium  so  as  to  be  able  to  advise  farmers  in  the  region  of  optimal 
varieties  of  agricultural  products. 

In  the  1979/80  season  at  the  "Veredelingsstation  Heverlee"  experi¬ 
ments  were  run  on  3  varieties  of  Winterwheat  with  two  sets  of  treatments 
(with  fertilizers  and  without  fertilizers  Thus  we  have  two  sets  of 

k  *  3  populations  each  of  which  we  wish  to  rank  separately  on  yield  from 

2 

best,  second  best,  to  worst.  The  yields  are  put  on  a  kg/m  basis  as 

2 

plots  are  of  varying  sizes,  i.e.  either  15.12  or  11. 34  m  (but  each 

2 

plot  received  300  seeds/m  ).  The  data  obtained  are  given  in  Table  4 
below.  (Thanks  for  this  dataset  are  due  to  ir.  J.  Niclaes  and  Dr. 

L.  Kempeneers  of  the  Veredelingsstation.) 

With  the  choice  of  Hq  =  15  observations,  we  are  interested  in  the 
following  questions. 

2  ... 

1.  If  5"  =■  .03  kg/m  is  a  difference  of  basic  interest  (while  interchanges 

in  ordening  of  populations  closer  than  6::  in  their  means  are  not  of 

strong  importance),  with  Hq  •  15  what  P::  can  we  guarantee  of  a  fully 

correct  ordering  in  the  PPF  formulation  for  non-fertilized  treatments  ? 

For  fertilized  treatments  ? 

::  The  fertilizers  which  were  used  are  :  100  units/ha  Nitrogen,  1.5  1/ha 
Chloormequatchloride ,  Herbicides  (Metabenzthiazuron  3  kg/ha  and  Mecoprop  + 
Ioxynil  4  1/ha) ,  and  Fungicides  (4  kg/ha  Spuitzwavel,  0.4  kg/ha  Benlate, 

3  kg/ha  Thiof anaat-methyl +maneb ,  and  again  2.5  kg/ha  Spuitzwavel). 
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Table  A.  Agricultural  Yields  (kg/m“),  at  two  substations  of  the  "Veredelings 
station  Heverlee",  1980 


— 

Vnrietv  1 

Variety  2 

! 

Variety  3 

*  * 

Number 

tf/0  Fert. 

Fert . 

W70  F.rt . 

Fert . 

W/O  Fert . 

F*rt.  j 

1 

.56  2:;:: 

.754 

.  575:": 

.780 

.  602:::: 

i 

.827  { 

* 

.648"“ 

.794 

.609"“ 

.800 

.  589;::: 

.813  1 

3 

.549:::: 

.787 

.615 

.800 

.  542 ::I! 

.  774  j 

A 

.516::;; 

.  708 

.  5 16:n: 

.800 

-  595 

.  767 

5 

.503 

.  721 

.485 

.761 

.467 

.  787  j 

6 

.538 

.734 

.564 

.721 

.512 

.708 

7 

.4  76 

.747 

.538 

.728 

.512 

.747 

8 

.467 

.741 

.450 

.  714 

.485 

.622 

9 

.503 

.701 

.582 

.721 

.520 

.  734 

10 

.582 

.694 

.494 

.681 

.503 

.668 

II 

.485 

.  741 

.529 

.  708 

.5  38 

.  728 

! 12 

.538 

.  721 

.600 

.668 

.529 

.694 

mm 

.46  7 

.406 

.467 

mm 

.511 

.520 

.414 

■ 

.503 

.582 

.547 

H 

.467 

.600 

.458 

mm 

.533 

.  740 

.53a 

.739 

.521 

.  743 

n 

.051 

.030 

.057 

.063 

.056 

.053 

In  order  to  take  only  one  more  observation  in  each  group,  it  follows 
from  (3)  that  we  have  to  look  for  the  largest  possible  h  value  such  that 


(  max  h2/6”2  <  16.  (15) 

\i-l,2,3  x' 


In  the  case  of  treatments  without  fertilizer  this  means  that 

2  2,2 

h  =  (.03)  1 6 / ( . 05 7)  .  Then  from  the  Monte  Carlo  tabulation  of  P(CR)  as  a 
function  of  h  (which  was  obtained  in  the  course  of  constructing  the  tables 


::::  Data  accompanied  with  are  obtained  from  a  11.34  m"  plot.  All  other 

2 

data  come  from  an  15.12  m“  plot. 


of  Section  5)  we  found  that  P::  »  .8340  can  be  guaranteed  of  a  fully  correct 
ordering  in  the  PPF  formulation.  Similarly,  in  the  case  of  treatments  with 
fertilizer  P"  =  .8531  can  be  guaranteed,  assuming  16  observations  for  each 
population  are  available. 

Moreover,  in  cases  of  no  fertilizer,  analysis  based  on  the  procedure 

described  in  Section  2  implies  that  one  can  be  83,40  Z  sure  that  a  ranking 

where  variety  2  gives  most  yield,  variety  1  second  most,  and  variety 

3  least,  is  correct  up  to  interchanges  of  the  varieties  whose  true 

2 

means  differ  by  6"  =  300  kg/ha  «  .03  kg/m  or  less.  In  the  cases  of  ferti¬ 
lizer  no  such  conclusion  can  be  made  because  of  the  missing  16th  observa¬ 
tion  for  variety  2  (though  a  similar  analysis  could  be  made  using  nQ  »  14). 

2.  WithP::»  .95  and  nQ  -  15,  what  6::  can  be  guaranteed  for  the  PPF  formu¬ 

lation  for  non-fertilized  treatments  ?  For  fertilized  treatments  ? 

From  (3)  again,  we  have  to  look  for  the  smallest  possible  <5::  value 
such  that 

/  max  s?)  (2.98)2/5"2  «  16  (16) 

U-1,2,3  V 

where  the  h  value  2.98  follows  from  the  tables.  For  the  case  of  no  ferti¬ 
lizer  this  gives  <$"  »  .0425,  while  in  the  case  of  treatment  with  fertili¬ 
zer  5”  *  .0395  can  be  guaranteed. 

..  2 

3.  If  6"  ■  ,03  kg/m  and  P--  =  .95,  which  number  of  observations,  additional 
to  Qq  -  15, should  be  used  in  future  trials  so  as  to  be  able  to  make 
appropriate  guarantees  on  ordering  analysis  in  the  PPF  formulation  ? 


To  find  the  required  number  of  additional  abservations ,  the  value 
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!  in,  s*)<2-98>2 

!  (.03)2 


0  7) 


is  to  be  calculated.  This  gives  in  the  no-fcrti lizer  treatment  case  33-15  *  18 

additional  observations , and  for  the  data  with  fertilizing  treatment 

28-15  *  13  additional  observations.  Thus,  in  future  trials  the 

experimenter  is  recommended  to  design  the  experiment  with  a  least  33 

plots  per  variety  in  the  fertilizer  treatment  case  (28  in  the  no-ferti- 

lizer  treatment  case)  when  6"  is  chosen  .03  kg/m  and  P:;  is  to  be  .95. 
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5.  TABLES  NEEDED  FOR  IMPLEMENTATION 


Table  5.  The  Solution  h  of  Equation  (8)  for  :  k  “  2(1)25;  P"  ■  .75(.05).95, 
.975,  .99;  nQ  =■  2(1)10(5)30.® 

It  •  2 


Cont*  Table 


k  -  1 1 


24 


I 

I 


Cont.  Table  5 

k  -  14 


\n0 

F:: 

2 

3 

4 

5 

6 

7 

8 

Q 

10 

15 

20 

25 

30 

”  i 

i 

.75 

29.7 

7.1 

4.9 

4.2 

3.8 

3.6 

3.4 

3. 79 

3.13 

3.07 

2.99 

2.98 

j 

2.88  j 

.$0 

31 

8.0 

5.3 

4.5 

4.  I 

3.8 

3.6 

3.47 

3.31 

3.22 

3.15 

3.  14 

3.03  i 

.85 

50 

9.3 

5.9 

4.9 

4.4 

4.1 

3.8 

3.  71 

3.52 

3.43 

3.36 

3.32 

3.  19 

.90 

8! 

1  1.5 

6.8 

5.5 

4.8 

4.5 

4.2 

4.02 

3.78 

3.70 

3.61 

3.57 

3.93 

.95 

170 

16 

8.4 

6.6 

5.7 

5.  1 

4.9 

4.7 

4.57 

4.22 

4.13 

4.05 

3.96 

3.  78 

.975 

310 

22 

10.7 

8.0 

6.  7 

5.9 

5.5 

5.J 

5.12 

4.65 

4.57 

« .  42 

4.33 

4.09 

.99 

700 

36 

14 

10 

8 

7 

6 

6.0 

5.9 

5.2 

4.9 

4.8 

4.3 

4.5 

1 

k 


15 


B 

3 

4 

5 

6 

7 

3 

9 

10 

15 

20 

25 

30 

- 

7.3 

5.0 

4.3 

3.9 

3.7 

3.5 

3.4 

3.33 

3.18 

3.1! 

3.09 

3.03 

2.92  J 

.80 

41 

8.2 

5.4 

4.6 

4.1 

3.  * 

1.7 

3.6 

3.52 

3.  36 

3.26 

3.20 

3.  19 

3.06  1 

.35 

55 

9.6 

6.0 

5.0 

4.5 

4.2 

4.0 

3.9 

3.76 

3.57 

3.47 

3.90 

3.37 

3.23 

.90 

86 

12.0 

7.0 

5.6 

4.9 

4.6 

4.4 

4.2 

4.08 

3.83 

3.73 

3.67 

3-60 

3.46 

.95 

175 

1  7 

8.7 

6.7 

5.8 

5.2 

5.0 

4.8 

4.61 

4.26 

4.18 

4.09 

4.02 

3.81 

.975 

340 

23 

10.8 

b.2 

6.8 

5.9 

5.6 

5.3 

5.19 

4.68 

4.60 

4,44 

4.39 

4.13 

.99 

37 

15 

10 

8 

7 

6 

6.0 

6.0 

5.2 

4.9 

4.8 

4.8 

4.5 

k  -  16 
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6.  CONCLUSIONS  AND  RECOMMENDATIONS 

In  this  paper  we  have  given  tables  needed  to  implement  new  procedures 
for  ranking  a  set  of  k(  >  2)  independent  normal  populations  with  unknown 
means  and  variances.  The  procedure  is  based  on  two-stage  sampling  from 
each  population,  whereby  the  same  initial  sample  size  n^  ( >  2)  is  taken 
from  each  population.  The  examples  provided  illustrate  the  wide  appli¬ 
cability  of  this  procedure,  point  out  the  various  steps  involved  in 
carrying  out  the  procedures,  and  indicate  proper  use  of  the  tables.  The 
tables  are  believed  to  be  accurate  within  error  of  a  Monte  Carlo  procedure 
using  10,000  samples.  The  tables  should  be  helpful  to  any  experimenter 
who  wishes  to  design  and  carry  out  his  experiment  with  the  goal  of 
correctly  ranking  several  populations  with  a  high  probability  of  being 
correct.  Ranking  procedures  in  an  analysis  of  variance  setting  are 
receiving  increasing  emphasis  in  the  field  of  ranking  and  selection. 
Procedures  for  analysis  of  the  data  of  example  2  supplemented  with  data 
from  other  locations  as  well  are  currently  under  development. 
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